{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d95f841a-63c9-41d4-aea1-496b3d2024dd",
   "metadata": {},
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "补充代码来自 <a href=\"http://mng.bz/orYv\">从零构建大型语言模型</a> 一书，作者为 <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>代码仓库: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25aa40e3-5109-433f-9153-f5770531fe94",
   "metadata": {},
   "source": [
    "# 第2章：处理文本"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76d5d2c0-cba8-404e-9bf3-71a218cae3cf",
   "metadata": {},
   "source": [
    "本笔记本中使用的包如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "4d1305cf-12d5-46fe-a2c9-36fb71c5b3d3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-09-29T12:04:17.965756Z",
     "start_time": "2024-09-29T12:04:17.958088Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch 版本: 2.2.2\n",
      "tiktoken 版本: 0.6.0\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "# 打印torch库的版本\n",
    "print(\"torch 版本:\", version(\"torch\"))\n",
    "# 打印tiktoken库的版本\n",
    "print(\"tiktoken 版本:\", version(\"tiktoken\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a42fbfd-e3c2-43c2-bc12-f5f870a0b10a",
   "metadata": {},
   "source": [
    "- 本章涵盖数据准备和采样，以使输入数据为大型语言模型（LLM）做好“准备”"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "628b2922-594d-4ff9-bd82-04f1ebdf41f5",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/01.webp?timestamp=1\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2417139b-2357-44d2-bd67-23f5d7f52ae7",
   "metadata": {},
   "source": [
    "## 2.1 理解词嵌入"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b6816ae-e927-43a9-b4dd-e47a9b0e1cf6",
   "metadata": {},
   "source": [
    "- 本节无代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f69dab7-a433-427a-9e5b-b981062d6296",
   "metadata": {},
   "source": [
    "- 嵌入形式有很多种；本书中我们专注于文本嵌入"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba08d16f-f237-4166-bf89-0e9fe703e7b4",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/02.webp\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "288c4faf-b93a-4616-9276-7a4aa4b5e9ba",
   "metadata": {},
   "source": [
    "- 大型语言模型（LLM）处理的是高维空间中的嵌入（即数千维）\n",
    "- 由于我们无法可视化如此高维的空间（人类的思维通常局限于1维、2维或3维），下图展示了一个二维的嵌入空间"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6b80160-1f10-4aad-a85e-9c79444de9e6",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/03.webp\" width=\"300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eddbb984-8d23-40c5-bbfa-c3c379e7eec3",
   "metadata": {},
   "source": [
    "## 2.2 文本的分词处理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9c90731-7dc9-4cd3-8c4a-488e33b48e80",
   "metadata": {},
   "source": [
    "- 在本节中，我们将文本进行分词处理，也就是将文本拆分为更小的单元，比如单个单词和标点符号"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09872fdb-9d4e-40c4-949d-52a01a43ec4b",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/04.webp\" width=\"300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cceaa18-833d-46b6-b211-b20c53902805",
   "metadata": {},
   "source": [
    "- 加载我们要处理的原始文本\n",
    "- [《裁决》 作者：伊迪丝·华顿](https://en.wikisource.org/wiki/The_Verdict) 是一篇公共领域的短篇小说"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "40f9d9b1-6d32-485a-825a-a95392a86d79",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib.request\n",
    "\n",
    "# 检查文件是否已存在，如果不存在则下载文件\n",
    "if not os.path.exists(\"the-verdict.txt\"):\n",
    "    url = (\"https://raw.githubusercontent.com/rasbt/\"\n",
    "           \"LLMs-from-scratch/main/ch02/01_main-chapter-code/\"\n",
    "           \"the-verdict.txt\")\n",
    "    file_path = \"the-verdict.txt\"\n",
    "    # 从指定URL下载文件并保存为\"the-verdict.txt\"\n",
    "    urllib.request.urlretrieve(url, file_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8a769e87-470a-48b9-8bdb-12841b416198",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "字符总数: 20479\n",
      "I HAD always thought Jack Gisburn rather a cheap genius--though a good fellow enough--so it was no \n"
     ]
    }
   ],
   "source": [
    "# 打开文件“the-verdict.txt”，读取其中的内容\n",
    "with open(\"the-verdict.txt\", \"r\", encoding=\"utf-8\") as f:\n",
    "    raw_text = f.read()\n",
    "\n",
    "# 打印文本的总字符数\n",
    "print(\"字符总数:\", len(raw_text))\n",
    "\n",
    "# 打印前99个字符的内容\n",
    "print(raw_text[:99])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b971a46-ac03-4368-88ae-3f20279e8f4e",
   "metadata": {},
   "source": [
    "- 目标是将此文本进行分词处理并嵌入到大型语言模型中\n",
    "- 让我们基于一些简单的示例文本开发一个简单的分词器，之后可以将其应用于上述文本\n",
    "- 以下正则表达式将基于空白字符进行分割"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "737dd5b0-9dbb-4a97-9ae4-3482c8c04be7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hello,', ' ', 'world.', ' ', 'This,', ' ', 'is', ' ', 'a', ' ', 'test.']\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "# 示例文本\n",
    "text = \"Hello, world. This, is a test.\"\n",
    "\n",
    "# 使用正则表达式基于空白字符进行分割\n",
    "result = re.split(r'(\\s)', text)\n",
    "\n",
    "# 打印分词后的结果\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8c40c18-a9d5-4703-bf71-8261dbcc5ee3",
   "metadata": {},
   "source": [
    "- 我们不仅要基于空白字符进行分割，还需要对逗号和句号进行分割，因此我们将修改正则表达式以实现该功能"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ea02489d-01f9-4247-b7dd-a0d63f62ef07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hello', ',', '', ' ', 'world', '.', '', ' ', 'This', ',', '', ' ', 'is', ' ', 'a', ' ', 'test', '.', '']\n"
     ]
    }
   ],
   "source": [
    "# 修改正则表达式以同时基于逗号、句号和空白字符进行分割\n",
    "result = re.split(r'([,.]|\\s)', text)\n",
    "\n",
    "# 打印分割后的结果\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "461d0c86-e3af-4f87-8fae-594a9ca9b6ad",
   "metadata": {},
   "source": [
    "- 正如我们所见，这会生成一些空字符串，让我们将它们移除"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4d8a6fb7-2e62-4a12-ad06-ccb04f25fed7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hello', ',', 'world', '.', 'This', ',', 'is', 'a', 'test', '.']\n"
     ]
    }
   ],
   "source": [
    "# 去除每个项中的空白字符，然后过滤掉任何空字符串\n",
    "result = [item for item in result if item.strip()]\n",
    "\n",
    "# 打印过滤后的结果\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "250e8694-181e-496f-895d-7cb7d92b5562",
   "metadata": {},
   "source": [
    "- 这看起来不错，但我们还需要处理其他类型的标点符号，例如句号、问号等"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ed3a9467-04b4-49d9-96c5-b8042bcf8374",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hello', ',', 'world', '.', 'Is', 'this', '--', 'a', 'test', '?']\n"
     ]
    }
   ],
   "source": [
    "# 示例文本，包含多种标点符号\n",
    "text = \"Hello, world. Is this-- a test?\"\n",
    "\n",
    "# 正则表达式，处理逗号、句号、分号、问号、感叹号、括号、引号、双破折号和空白字符\n",
    "result = re.split(r'([,.:;?_!\"()\\']|--|\\s)', text)\n",
    "\n",
    "# 去除空白并过滤掉空字符串\n",
    "result = [item.strip() for item in result if item.strip()]\n",
    "\n",
    "# 打印结果\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bbea70b-c030-45d9-b09d-4318164c0bb4",
   "metadata": {},
   "source": [
    "- 这已经非常不错了，现在我们可以将这个分词方法应用到原始文本中了"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cbe9330-b587-4262-be9f-497a84ec0e8a",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/05.webp\" width=\"350px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8c567caa-8ff5-49a8-a5cc-d365b0a78a99",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['I', 'HAD', 'always', 'thought', 'Jack', 'Gisburn', 'rather', 'a', 'cheap', 'genius', '--', 'though', 'a', 'good', 'fellow', 'enough', '--', 'so', 'it', 'was', 'no', 'great', 'surprise', 'to', 'me', 'to', 'hear', 'that', ',', 'in']\n"
     ]
    }
   ],
   "source": [
    "# 对原始文本应用分词处理\n",
    "preprocessed = re.split(r'([,.:;?_!\"()\\']|--|\\s)', raw_text)\n",
    "\n",
    "# 去除空白并过滤掉空字符串\n",
    "preprocessed = [item.strip() for item in preprocessed if item.strip()]\n",
    "\n",
    "# 打印前30个分词结果\n",
    "print(preprocessed[:30])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a19e1a-5105-4ddb-812a-b7d3117eab95",
   "metadata": {},
   "source": [
    "- 让我们计算分词后的总词元数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "35db7b5e-510b-4c45-995f-f5ad64a8e19c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4690\n"
     ]
    }
   ],
   "source": [
    "# 打印分词后的总词元数\n",
    "print(len(preprocessed))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b5ce8fe-3a07-4f2a-90f1-a0321ce3a231",
   "metadata": {},
   "source": [
    "## 2.3 将词元转换为词元ID"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5204973-f414-4c0d-87b0-cfec1f06e6ff",
   "metadata": {},
   "source": [
    "- 接下来，我们将文本词元转换为词元ID，之后可以通过嵌入层进行处理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "177b041d-f739-43b8-bd81-0443ae3a7f8d",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/06.webp\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5973794-7002-4202-8b12-0900cd779720",
   "metadata": {},
   "source": [
    "- 从这些词元中，我们现在可以构建一个包含所有唯一词元的词汇表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7fdf0533-5ab6-42a5-83fa-a3b045de6396",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1130\n"
     ]
    }
   ],
   "source": [
    "# 对所有词元去重并排序，构建词汇表\n",
    "all_words = sorted(set(preprocessed))\n",
    "\n",
    "# 计算词汇表的大小\n",
    "vocab_size = len(all_words)\n",
    "\n",
    "# 打印词汇表大小\n",
    "print(vocab_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "77d00d96-881f-4691-bb03-84fec2a75a26",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将词汇表中的每个词元映射为唯一的整数ID\n",
    "vocab = {token: integer for integer, token in enumerate(all_words)}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75bd1f81-3a8f-4dd9-9dd6-e75f32dacbe3",
   "metadata": {},
   "source": [
    "- 下面是词汇表中的前50个条目："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e1c5de4a-aa4e-4aec-b532-10bb364039d6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('!', 0)\n",
      "('\"', 1)\n",
      "(\"'\", 2)\n",
      "('(', 3)\n",
      "(')', 4)\n",
      "(',', 5)\n",
      "('--', 6)\n",
      "('.', 7)\n",
      "(':', 8)\n",
      "(';', 9)\n",
      "('?', 10)\n",
      "('A', 11)\n",
      "('Ah', 12)\n",
      "('Among', 13)\n",
      "('And', 14)\n",
      "('Are', 15)\n",
      "('Arrt', 16)\n",
      "('As', 17)\n",
      "('At', 18)\n",
      "('Be', 19)\n",
      "('Begin', 20)\n",
      "('Burlington', 21)\n",
      "('But', 22)\n",
      "('By', 23)\n",
      "('Carlo', 24)\n",
      "('Chicago', 25)\n",
      "('Claude', 26)\n",
      "('Come', 27)\n",
      "('Croft', 28)\n",
      "('Destroyed', 29)\n",
      "('Devonshire', 30)\n",
      "('Don', 31)\n",
      "('Dubarry', 32)\n",
      "('Emperors', 33)\n",
      "('Florence', 34)\n",
      "('For', 35)\n",
      "('Gallery', 36)\n",
      "('Gideon', 37)\n",
      "('Gisburn', 38)\n",
      "('Gisburns', 39)\n",
      "('Grafton', 40)\n",
      "('Greek', 41)\n",
      "('Grindle', 42)\n",
      "('Grindles', 43)\n",
      "('HAD', 44)\n",
      "('Had', 45)\n",
      "('Hang', 46)\n",
      "('Has', 47)\n",
      "('He', 48)\n",
      "('Her', 49)\n",
      "('Hermia', 50)\n"
     ]
    }
   ],
   "source": [
    "# 打印词汇表中的前50个条目\n",
    "for i, item in enumerate(vocab.items()):\n",
    "    print(item)\n",
    "    if i >= 50:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b1dc314-351b-476a-9459-0ec9ddc29b19",
   "metadata": {},
   "source": [
    "- 下面，我们用一个小词汇表来演示对一个简短示例文本的分词处理："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67407a9f-0202-4e7c-9ed7-1b3154191ebc",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/07.webp?123\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e569647-2589-4c9d-9a5c-aef1c88a0a9a",
   "metadata": {},
   "source": [
    "- 现在将所有内容整合到一个分词器类中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f531bf46-7c25-4ef8-bff8-0d27518676d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleTokenizerV1:\n",
    "    def __init__(self, vocab):\n",
    "        # 初始化时，将词汇表映射为词元字符串到整数ID的映射\n",
    "        self.str_to_int = vocab\n",
    "        # 创建整数ID到词元字符串的反向映射\n",
    "        self.int_to_str = {i: s for s, i in vocab.items()}\n",
    "    \n",
    "    def encode(self, text):\n",
    "        # 使用正则表达式对输入文本进行分词\n",
    "        preprocessed = re.split(r'([,.:;?_!\"()\\']|--|\\s)', text)\n",
    "                                \n",
    "        # 去除空白字符并过滤掉空字符串\n",
    "        preprocessed = [\n",
    "            item.strip() for item in preprocessed if item.strip()\n",
    "        ]\n",
    "        # 将分词结果转换为词元ID\n",
    "        ids = [self.str_to_int[s] for s in preprocessed]\n",
    "        return ids\n",
    "        \n",
    "    def decode(self, ids):\n",
    "        # 将词元ID转换为原始文本\n",
    "        text = \" \".join([self.int_to_str[i] for i in ids])\n",
    "        # 在标点符号前去除空格\n",
    "        text = re.sub(r'\\s+([,.?!\"()\\'])', r'\\1', text)\n",
    "        return text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dee7a1e5-b54f-4ca1-87ef-3d663c4ee1e7",
   "metadata": {},
   "source": [
    "- `encode` 函数将文本转换为词元ID\n",
    "- `decode` 函数将词元ID转换回文本"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc21d347-ec03-4823-b3d4-9d686e495617",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/08.webp?123\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2950a94-6b0d-474e-8ed0-66d0c3c1a95c",
   "metadata": {},
   "source": [
    "- 我们可以使用分词器将文本编码（即分词）为整数\n",
    "- 这些整数之后可以作为嵌入，作为大型语言模型（LLM）的输入"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "647364ec-7995-4654-9b4a-7607ccf5f1e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 56, 2, 850, 988, 602, 533, 746, 5, 1126, 596, 5, 1, 67, 7, 38, 851, 1108, 754, 793, 7]\n"
     ]
    }
   ],
   "source": [
    "# 使用词汇表初始化分词器\n",
    "tokenizer = SimpleTokenizerV1(vocab)\n",
    "\n",
    "# 示例文本\n",
    "text = \"\"\"\"It's the last he painted, you know,\" \n",
    "           Mrs. Gisburn said with pardonable pride.\"\"\"\n",
    "\n",
    "# 将文本编码为词元ID\n",
    "ids = tokenizer.encode(text)\n",
    "\n",
    "# 打印词元ID\n",
    "print(ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3201706e-a487-4b60-b99d-5765865f29a0",
   "metadata": {},
   "source": [
    "- 我们可以将这些整数解码回文本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "01d8c8fb-432d-4a49-b332-99f23b233746",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\" It\\' s the last he painted, you know,\" Mrs. Gisburn said with pardonable pride.'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 将词元ID解码回文本\n",
    "tokenizer.decode(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "54f6aa8b-9827-412e-9035-e827296ab0fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\" It\\' s the last he painted, you know,\" Mrs. Gisburn said with pardonable pride.'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 将文本编码为词元ID后，再将其解码回文本\n",
    "tokenizer.decode(tokenizer.encode(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b821ef8-4d53-43b6-a2b2-aef808c343c7",
   "metadata": {},
   "source": [
    "## 2.4 添加特殊的上下文词元"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "863d6d15-a3e2-44e0-b384-bb37f17cf443",
   "metadata": {},
   "source": [
    "- 为未知词元和标记文本结束添加一些“特殊”词元是非常有用的"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa7fc96c-e1fd-44fb-b7f5-229d7c7922a4",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/09.webp?123\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d709d57-2486-4152-b7f9-d3e4bd8634cd",
   "metadata": {},
   "source": [
    "- 一些分词器使用特殊词元来为大型语言模型（LLM）提供额外的上下文信息\n",
    "- 其中一些特殊词元包括：\n",
    "  - `[BOS]` （序列开始）用于标记文本的开始\n",
    "  - `[EOS]` （序列结束）用于标记文本的结束（通常用于连接多个不相关的文本，例如两篇不同的维基百科文章或两本不同的书籍，等等）\n",
    "  - `[PAD]` （填充）用于在训练LLM时批处理大小大于1的情况下，填充长度不同的文本，以使所有文本长度一致\n",
    "  - `[UNK]` 表示词汇表中未包含的词\n",
    "\n",
    "- 请注意，GPT-2 不需要上述的这些特殊词元，它只使用 `<|endoftext|>` 词元来简化处理\n",
    "- `<|endoftext|>` 类似于上述的 `[EOS]` 词元\n",
    "- GPT 也使用 `<|endoftext|>` 作为填充词元（因为我们通常在训练批量输入时使用掩码，不会关注填充的词元，所以这些词元具体是什么并不重要）\n",
    "- GPT-2 不使用 `<UNK>` 词元来表示词汇表外的词汇；相反，GPT-2 使用字节对编码（BPE）分词器，它将词汇分解为子词单位，关于这一点我们将在后面的章节中讨论"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a336b43b-7173-49e7-bd80-527ad4efb271",
   "metadata": {},
   "source": [
    "- 我们在两个独立的文本来源之间使用 `<|endoftext|>` 词元："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52442951-752c-4855-9752-b121a17fef55",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/10.webp\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c661a397-da06-4a86-ac27-072dbe7cb172",
   "metadata": {},
   "source": [
    "- 让我们看看对以下文本进行分词会发生什么："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d5767eff-440c-4de1-9289-f789349d6b85",
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'Hello'",
     "output_type": "error",
     "traceback": [
      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[0;31mKeyError\u001B[0m                                  Traceback (most recent call last)",
      "Cell \u001B[0;32mIn[19], line 8\u001B[0m\n\u001B[1;32m      5\u001B[0m text \u001B[38;5;241m=\u001B[39m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mHello, do you like tea. Is this-- a test?\u001B[39m\u001B[38;5;124m\"\u001B[39m\n\u001B[1;32m      7\u001B[0m \u001B[38;5;66;03m# 将文本编码为词元ID\u001B[39;00m\n\u001B[0;32m----> 8\u001B[0m \u001B[43mtokenizer\u001B[49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mencode\u001B[49m\u001B[43m(\u001B[49m\u001B[43mtext\u001B[49m\u001B[43m)\u001B[49m\n",
      "Cell \u001B[0;32mIn[15], line 17\u001B[0m, in \u001B[0;36mSimpleTokenizerV1.encode\u001B[0;34m(self, text)\u001B[0m\n\u001B[1;32m     13\u001B[0m preprocessed \u001B[38;5;241m=\u001B[39m [\n\u001B[1;32m     14\u001B[0m     item\u001B[38;5;241m.\u001B[39mstrip() \u001B[38;5;28;01mfor\u001B[39;00m item \u001B[38;5;129;01min\u001B[39;00m preprocessed \u001B[38;5;28;01mif\u001B[39;00m item\u001B[38;5;241m.\u001B[39mstrip()\n\u001B[1;32m     15\u001B[0m ]\n\u001B[1;32m     16\u001B[0m \u001B[38;5;66;03m# 将分词结果转换为词元ID\u001B[39;00m\n\u001B[0;32m---> 17\u001B[0m ids \u001B[38;5;241m=\u001B[39m [\u001B[38;5;28mself\u001B[39m\u001B[38;5;241m.\u001B[39mstr_to_int[s] \u001B[38;5;28;01mfor\u001B[39;00m s \u001B[38;5;129;01min\u001B[39;00m preprocessed]\n\u001B[1;32m     18\u001B[0m \u001B[38;5;28;01mreturn\u001B[39;00m ids\n",
      "Cell \u001B[0;32mIn[15], line 17\u001B[0m, in \u001B[0;36m<listcomp>\u001B[0;34m(.0)\u001B[0m\n\u001B[1;32m     13\u001B[0m preprocessed \u001B[38;5;241m=\u001B[39m [\n\u001B[1;32m     14\u001B[0m     item\u001B[38;5;241m.\u001B[39mstrip() \u001B[38;5;28;01mfor\u001B[39;00m item \u001B[38;5;129;01min\u001B[39;00m preprocessed \u001B[38;5;28;01mif\u001B[39;00m item\u001B[38;5;241m.\u001B[39mstrip()\n\u001B[1;32m     15\u001B[0m ]\n\u001B[1;32m     16\u001B[0m \u001B[38;5;66;03m# 将分词结果转换为词元ID\u001B[39;00m\n\u001B[0;32m---> 17\u001B[0m ids \u001B[38;5;241m=\u001B[39m [\u001B[38;5;28;43mself\u001B[39;49m\u001B[38;5;241;43m.\u001B[39;49m\u001B[43mstr_to_int\u001B[49m\u001B[43m[\u001B[49m\u001B[43ms\u001B[49m\u001B[43m]\u001B[49m \u001B[38;5;28;01mfor\u001B[39;00m s \u001B[38;5;129;01min\u001B[39;00m preprocessed]\n\u001B[1;32m     18\u001B[0m \u001B[38;5;28;01mreturn\u001B[39;00m ids\n",
      "\u001B[0;31mKeyError\u001B[0m: 'Hello'"
     ]
    }
   ],
   "source": [
    "# 使用词汇表初始化分词器\n",
    "tokenizer = SimpleTokenizerV1(vocab)\n",
    "\n",
    "# 示例文本\n",
    "text = \"Hello, do you like tea. Is this-- a test?\"\n",
    "\n",
    "# 将文本编码为词元ID\n",
    "tokenizer.encode(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc53ee0c-fe2b-4cd8-a946-5471f7651acf",
   "metadata": {},
   "source": [
    "- 上述代码产生了错误，因为单词 \"Hello\" 不在词汇表中\n",
    "- 为了解决这种情况，我们可以在词汇表中添加像 `\"<|unk|>\"` 这样的特殊词元来表示未知单词\n",
    "- 既然我们已经在扩展词汇表了，那就再添加一个名为 `\"<|endoftext|>\"` 的词元，这是在 GPT-2 训练中用于标记文本结束的词元（在连接的文本之间也使用，比如当我们的训练数据集中包含多篇文章、书籍等时）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ce9df29c-6c5b-43f1-8c1a-c7f7b79db78f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 对所有词元去重并排序，构建词汇表\n",
    "all_tokens = sorted(list(set(preprocessed)))\n",
    "\n",
    "# 扩展词汇表，添加特殊词元 \"<|endoftext|>\" 和 \"<|unk|>\"\n",
    "all_tokens.extend([\"<|endoftext|>\", \"<|unk|>\"])\n",
    "\n",
    "# 将词汇表中的每个词元映射为唯一的整数ID\n",
    "vocab = {token: integer for integer, token in enumerate(all_tokens)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "57c3143b-e860-4d3b-a22a-de22b547a6a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1132"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 打印词汇表中的词元总数\n",
    "len(vocab.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "50e51bb1-ae05-4aa8-a9ff-455b65ed1959",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('younger', 1127)\n",
      "('your', 1128)\n",
      "('yourself', 1129)\n",
      "('<|endoftext|>', 1130)\n",
      "('<|unk|>', 1131)\n"
     ]
    }
   ],
   "source": [
    "# 打印词汇表中最后5个词元及其对应的ID\n",
    "for i, item in enumerate(list(vocab.items())[-5:]):\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1daa2b0-6e75-412b-ab53-1f6fb7b4d453",
   "metadata": {},
   "source": [
    "- 我们还需要相应地调整分词器，使其知道何时以及如何使用新的 `<unk>` 词元"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "948861c5-3f30-4712-a234-725f20d26f68",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleTokenizerV2:\n",
    "    def __init__(self, vocab):\n",
    "        # 初始化时，将词汇表映射为词元字符串到整数ID的映射\n",
    "        self.str_to_int = vocab\n",
    "        # 创建整数ID到词元字符串的反向映射\n",
    "        self.int_to_str = {i: s for s, i in vocab.items()}\n",
    "    \n",
    "    def encode(self, text):\n",
    "        # 使用正则表达式对输入文本进行分词\n",
    "        preprocessed = re.split(r'([,.:;?_!\"()\\']|--|\\s)', text)\n",
    "        # 去除空白字符并过滤掉空字符串\n",
    "        preprocessed = [item.strip() for item in preprocessed if item.strip()]\n",
    "        # 如果词元不在词汇表中，使用 \"<|unk|>\" 作为未知词元\n",
    "        preprocessed = [\n",
    "            item if item in self.str_to_int \n",
    "            else \"<|unk|>\" for item in preprocessed\n",
    "        ]\n",
    "        # 将分词结果转换为词元ID\n",
    "        ids = [self.str_to_int[s] for s in preprocessed]\n",
    "        return ids\n",
    "        \n",
    "    def decode(self, ids):\n",
    "        # 将词元ID转换为原始文本\n",
    "        text = \" \".join([self.int_to_str[i] for i in ids])\n",
    "        # 在标点符号前去除空格\n",
    "        text = re.sub(r'\\s+([,.:;?!\"()\\'])', r'\\1', text)\n",
    "        return text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa728dd1-9d35-4ac7-938f-d411d73083f6",
   "metadata": {},
   "source": [
    "让我们使用修改后的分词器尝试对文本进行分词："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4133c502-18ac-4412-9f43-01caf4efa3dc",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'SimpleTokenizerV2' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[0;31mNameError\u001B[0m                                 Traceback (most recent call last)",
      "Cell \u001B[0;32mIn[15], line 2\u001B[0m\n\u001B[1;32m      1\u001B[0m \u001B[38;5;66;03m# 使用更新后的分词器\u001B[39;00m\n\u001B[0;32m----> 2\u001B[0m tokenizer \u001B[38;5;241m=\u001B[39m \u001B[43mSimpleTokenizerV2\u001B[49m(vocab)\n\u001B[1;32m      4\u001B[0m \u001B[38;5;66;03m# 两段示例文本\u001B[39;00m\n\u001B[1;32m      5\u001B[0m text1 \u001B[38;5;241m=\u001B[39m \u001B[38;5;124m\"\u001B[39m\u001B[38;5;124mHello, do you like tea?\u001B[39m\u001B[38;5;124m\"\u001B[39m\n",
      "\u001B[0;31mNameError\u001B[0m: name 'SimpleTokenizerV2' is not defined"
     ]
    }
   ],
   "source": [
    "# 使用更新后的分词器\n",
    "tokenizer = SimpleTokenizerV2(vocab)\n",
    "\n",
    "# 两段示例文本\n",
    "text1 = \"Hello, do you like tea?\"\n",
    "text2 = \"In the sunlit terraces of the palace.\"\n",
    "\n",
    "# 使用 \"<|endoftext|>\" 将两段文本连接起来\n",
    "text = \" <|endoftext|> \".join((text1, text2))\n",
    "\n",
    "# 打印连接后的文本\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "7ed395fe-dc1b-4ed2-b85b-457cc35aab60",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1131, 5, 355, 1126, 628, 975, 10, 1130, 55, 988, 956, 984, 722, 988, 1131, 7]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 对连接后的文本进行编码（分词）\n",
    "tokenizer.encode(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "059367f9-7a60-4c0d-8a00-7c4c766d0ebc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<|unk|>, do you like tea? <|endoftext|> In the sunlit terraces of the <|unk|>.'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 对文本进行编码并解码回原始文本\n",
    "tokenizer.decode(tokenizer.encode(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c4ba34b-170f-4e71-939b-77aabb776f14",
   "metadata": {},
   "source": [
    "## 2.5 字节对编码 (BytePair Encoding)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2309494c-79cf-4a2d-bc28-a94d602f050e",
   "metadata": {},
   "source": [
    "- GPT-2 使用了字节对编码（BPE）作为其分词器\n",
    "- 它允许模型将不在其预定义词汇表中的单词分解为更小的子词单元甚至是单个字符，从而能够处理词汇表外的单词\n",
    "- 例如，如果 GPT-2 的词汇表中没有“unfamiliarword”这个词，它可能会将其分词为 [\"unfam\", \"iliar\", \"word\"]，或根据其训练的 BPE 合并规则进行其他子词拆分\n",
    "- 原始的 BPE 分词器可以在这里找到：[https://github.com/openai/gpt-2/blob/master/src/encoder.py](https://github.com/openai/gpt-2/blob/master/src/encoder.py)\n",
    "- 在本章中，我们使用来自 OpenAI 开源库 [tiktoken](https://github.com/openai/tiktoken) 的 BPE 分词器，它的核心算法使用 Rust 实现，以提高计算性能\n",
    "- 我创建了一个笔记本，在 [./bytepair_encoder](../02_bonus_bytepair-encoder) 中对这两种实现进行了比较（在样本文本上，tiktoken 的速度快了约 5 倍）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "ede1d41f-934b-4bf4-8184-54394a257a94",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 使用以下命令安装 tiktoken 库\n",
    "# pip install tiktoken"
   ]
  },
  {
   "cell_type": "code",
   "id": "48967a77-7d17-42bf-9e92-fc619d63a59e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-09-29T12:07:09.295110Z",
     "start_time": "2024-09-29T12:07:09.282326Z"
    }
   },
   "source": [
    "import importlib\n",
    "import tiktoken\n",
    "\n",
    "# 打印 tiktoken 库的版本\n",
    "print(\"tiktoken 版本:\", importlib.metadata.version(\"tiktoken\"))"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tiktoken 版本: 0.6.0\n"
     ]
    }
   ],
   "execution_count": 1
  },
  {
   "cell_type": "code",
   "id": "6ad3312f-a5f7-4efc-9d7d-8ea09d7b5128",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-09-29T12:07:10.087373Z",
     "start_time": "2024-09-29T12:07:09.946685Z"
    }
   },
   "source": [
    "# 获取 GPT-2 的 BPE 分词器\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")"
   ],
   "outputs": [],
   "execution_count": 2
  },
  {
   "cell_type": "code",
   "id": "5ff2cd85-7cfb-4325-b390-219938589428",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-09-29T12:07:21.766058Z",
     "start_time": "2024-09-29T12:07:21.762602Z"
    }
   },
   "source": [
    "# 示例文本\n",
    "# text = (\n",
    "#     \"Hello, do you like tea? <|endoftext|> In the sunlit terraces\"\n",
    "#     \" of someunknownPlace.\"\n",
    "# )\n",
    "text = (\n",
    "    \"[0.2]\"\n",
    ")\n",
    "# 使用 tiktoken 分词器对文本进行编码，允许使用特殊词元 \"<|endoftext|>\"\n",
    "integers = tokenizer.encode(text, allowed_special={\"<|endoftext|>\"})\n",
    "\n",
    "# 打印编码后的整数ID\n",
    "print(integers)"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[58, 15, 13, 17, 60]\n"
     ]
    }
   ],
   "execution_count": 4
  },
  {
   "cell_type": "code",
   "id": "d26a48bb-f82e-41a8-a955-a1c9cf9d50ab",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-09-29T12:07:33.957863Z",
     "start_time": "2024-09-29T12:07:33.955034Z"
    }
   },
   "source": [
    "# 使用 tiktoken 分词器对整数ID进行解码\n",
    "strings = tokenizer.decode(integers)\n",
    "\n",
    "# 打印解码后的文本\n",
    "print(strings)"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.2]\n"
     ]
    }
   ],
   "execution_count": 5
  },
  {
   "cell_type": "markdown",
   "id": "e8c2e7b4-6a22-42aa-8e4d-901f06378d4a",
   "metadata": {},
   "source": [
    "- BPE 分词器将未知单词拆分为子词和单个字符："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c082d41f-33d7-4827-97d8-993d5a84bb3c",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/11.webp\" width=\"300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abbd7c0d-70f8-4386-a114-907e96c950b0",
   "metadata": {},
   "source": [
    "## 2.6 使用滑动窗口进行数据采样"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "509d9826-6384-462e-aa8a-a7c73cd6aad0",
   "metadata": {},
   "source": [
    "- 我们训练大型语言模型时，每次生成一个词，因此我们希望相应地准备训练数据，其中序列中的下一个词是要预测的目标："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39fb44f4-0c43-4a6a-9c2f-9cf31452354c",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/12.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "848d5ade-fd1f-46c3-9e31-1426e315c71b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5145\n"
     ]
    }
   ],
   "source": [
    "# 打开文件“the-verdict.txt”，读取其中的内容\n",
    "with open(\"the-verdict.txt\", \"r\", encoding=\"utf-8\") as f:\n",
    "    raw_text = f.read()\n",
    "\n",
    "# 使用 tiktoken 分词器对文本进行编码\n",
    "enc_text = tokenizer.encode(raw_text)\n",
    "\n",
    "# 打印编码后的词元总数\n",
    "print(len(enc_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cebd0657-5543-43ca-8011-2ae6bd0a5810",
   "metadata": {},
   "source": [
    "- 对于每个文本块，我们需要准备输入和目标\n",
    "- 由于我们希望模型预测下一个词，因此目标是将输入向右移动一个位置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e84424a7-646d-45b6-99e3-80d15fb761f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 从编码后的文本中获取一个样本，从第50个词元开始\n",
    "enc_sample = enc_text[50:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "dfbff852-a92f-48c8-a46d-143a0f109f40",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: [290, 4920, 2241, 287]\n",
      "y:      [4920, 2241, 287, 257]\n"
     ]
    }
   ],
   "source": [
    "# 设置上下文大小为4\n",
    "context_size = 4\n",
    "\n",
    "# 取出输入 x 为样本的前4个词元\n",
    "x = enc_sample[:context_size]\n",
    "\n",
    "# 取出目标 y 为样本向右偏移一个位置的4个词元\n",
    "y = enc_sample[1:context_size+1]\n",
    "\n",
    "# 打印输入 x 和目标 y\n",
    "print(f\"x: {x}\")\n",
    "print(f\"y:      {y}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "815014ef-62f7-4476-a6ad-66e20e42b7c3",
   "metadata": {},
   "source": [
    "- 逐步预测的过程如下所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "d97b031e-ed55-409d-95f2-aeb38c6fe366",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[290] ----> 4920\n",
      "[290, 4920] ----> 2241\n",
      "[290, 4920, 2241] ----> 287\n",
      "[290, 4920, 2241, 287] ----> 257\n"
     ]
    }
   ],
   "source": [
    "# 循环遍历，逐步打印每一步的输入上下文和对应的目标词元\n",
    "for i in range(1, context_size + 1):\n",
    "    context = enc_sample[:i]   # 输入上下文，长度从1到i\n",
    "    desired = enc_sample[i]    # 对应的目标词元\n",
    "\n",
    "    # 打印上下文及其目标\n",
    "    print(context, \"---->\", desired)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "f57bd746-dcbf-4433-8e24-ee213a8c34a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " and ---->  established\n",
      " and established ---->  himself\n",
      " and established himself ---->  in\n",
      " and established himself in ---->  a\n"
     ]
    }
   ],
   "source": [
    "# 循环遍历，逐步打印每一步的解码后的输入上下文和对应的目标词元\n",
    "for i in range(1, context_size + 1):\n",
    "    context = enc_sample[:i]    # 输入上下文，长度从1到i\n",
    "    desired = enc_sample[i]     # 对应的目标词元\n",
    "\n",
    "    # 打印解码后的上下文及其目标\n",
    "    print(tokenizer.decode(context), \"---->\", tokenizer.decode([desired]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "210d2dd9-fc20-4927-8d3d-1466cf41aae1",
   "metadata": {},
   "source": [
    "- 我们将在后面的章节中处理下一个词的预测，在此之前我们会讲解注意力机制\n",
    "- 现在，我们实现一个简单的数据加载器，它遍历输入数据集，并返回向右偏移一个位置的输入和目标"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1a1b47a-f646-49d1-bc70-fddf2c840796",
   "metadata": {},
   "source": [
    "- 安装并导入 PyTorch（安装提示请参见附录A）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "e1770134-e7f3-4725-a679-e04c3be48cac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PyTorch 版本: 2.2.2\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "# 打印 PyTorch 的版本\n",
    "print(\"PyTorch 版本:\", torch.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c9a3d50-885b-49bc-b791-9f5cc8bc7b7c",
   "metadata": {},
   "source": [
    "- 我们使用滑动窗口方法，位置移动 +1：\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/13.webp?123\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92ac652d-7b38-4843-9fbd-494cdc8ec12c",
   "metadata": {},
   "source": [
    "- 创建数据集和数据加载器，从输入文本数据集中提取块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "74b41073-4c9f-46e2-a1bd-d38e4122b375",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "# 定义 GPT 数据集类\n",
    "class GPTDatasetV1(Dataset):\n",
    "    def __init__(self, txt, tokenizer, max_length, stride):\n",
    "        self.input_ids = []\n",
    "        self.target_ids = []\n",
    "\n",
    "        # 对整个文本进行分词\n",
    "        token_ids = tokenizer.encode(txt, allowed_special={\"<|endoftext|>\"})\n",
    "\n",
    "        # 使用滑动窗口将文本分割为最大长度为 max_length 的重叠序列\n",
    "        for i in range(0, len(token_ids) - max_length, stride):\n",
    "            input_chunk = token_ids[i:i + max_length]\n",
    "            target_chunk = token_ids[i + 1: i + max_length + 1]\n",
    "            self.input_ids.append(torch.tensor(input_chunk))\n",
    "            self.target_ids.append(torch.tensor(target_chunk))\n",
    "\n",
    "    # 返回数据集的长度\n",
    "    def __len__(self):\n",
    "        return len(self.input_ids)\n",
    "\n",
    "    # 根据索引返回对应的输入和目标序列\n",
    "    def __getitem__(self, idx):\n",
    "        return self.input_ids[idx], self.target_ids[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "5eb30ebe-97b3-43c5-9ff1-a97d621b3c4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dataloader_v1(txt, batch_size=4, max_length=256, \n",
    "                         stride=128, shuffle=True, drop_last=True,\n",
    "                         num_workers=0):\n",
    "\n",
    "    # 初始化分词器\n",
    "    tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "\n",
    "    # 创建数据集\n",
    "    dataset = GPTDatasetV1(txt, tokenizer, max_length, stride)\n",
    "\n",
    "    # 创建数据加载器\n",
    "    dataloader = DataLoader(\n",
    "        dataset,\n",
    "        batch_size=batch_size,\n",
    "        shuffle=shuffle,\n",
    "        drop_last=drop_last,\n",
    "        num_workers=num_workers\n",
    "    )\n",
    "\n",
    "    return dataloader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42dd68ef-59f7-45ff-ba44-e311c899ddcd",
   "metadata": {},
   "source": [
    "- 让我们使用批量大小为1，且上下文大小为4的设置来测试数据加载器："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "df31d96c-6bfd-4564-a956-6192242d7579",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 打开文件“the-verdict.txt”，读取其中的内容\n",
    "with open(\"the-verdict.txt\", \"r\", encoding=\"utf-8\") as f:\n",
    "    raw_text = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "9226d00c-ad9a-4949-a6e4-9afccfc7214f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[tensor([[  40,  367, 2885, 1464]]), tensor([[ 367, 2885, 1464, 1807]])]\n"
     ]
    }
   ],
   "source": [
    "# 使用批量大小为1，上下文大小为4的设置创建数据加载器\n",
    "dataloader = create_dataloader_v1(\n",
    "    raw_text, batch_size=1, max_length=4, stride=1, shuffle=False\n",
    ")\n",
    "\n",
    "# 获取数据加载器的迭代器\n",
    "data_iter = iter(dataloader)\n",
    "\n",
    "# 获取第一个批次数据\n",
    "first_batch = next(data_iter)\n",
    "\n",
    "# 打印第一个批次的数据\n",
    "print(first_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "10deb4bc-4de1-4d20-921e-4b1c7a0e1a6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[tensor([[ 367, 2885, 1464, 1807]]), tensor([[2885, 1464, 1807, 3619]])]\n"
     ]
    }
   ],
   "source": [
    "# 获取第二个批次数据\n",
    "second_batch = next(data_iter)\n",
    "\n",
    "# 打印第二个批次的数据\n",
    "print(second_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b006212f-de45-468d-bdee-5806216d1679",
   "metadata": {},
   "source": [
    "- 如下所示，使用与上下文长度相等的滑动步长（此处为4）的示例："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cb467e0-bdcd-4dda-b9b0-a738c5d33ac3",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/14.webp\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1ae6d45-f26e-4b83-9c7b-cff55ffa7d16",
   "metadata": {},
   "source": [
    "- 我们还可以创建批处理输出\n",
    "- 请注意，这里我们增加了滑动步长，以避免批次之间的重叠，因为过多的重叠可能会导致过拟合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "1916e7a6-f03d-4f09-91a6-d0bdbac5a58c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inputs:\n",
      " tensor([[   40,   367,  2885,  1464],\n",
      "        [ 1807,  3619,   402,   271],\n",
      "        [10899,  2138,   257,  7026],\n",
      "        [15632,   438,  2016,   257],\n",
      "        [  922,  5891,  1576,   438],\n",
      "        [  568,   340,   373,   645],\n",
      "        [ 1049,  5975,   284,   502],\n",
      "        [  284,  3285,   326,    11]])\n",
      "\n",
      "Targets:\n",
      " tensor([[  367,  2885,  1464,  1807],\n",
      "        [ 3619,   402,   271, 10899],\n",
      "        [ 2138,   257,  7026, 15632],\n",
      "        [  438,  2016,   257,   922],\n",
      "        [ 5891,  1576,   438,   568],\n",
      "        [  340,   373,   645,  1049],\n",
      "        [ 5975,   284,   502,   284],\n",
      "        [ 3285,   326,    11,   287]])\n"
     ]
    }
   ],
   "source": [
    "# 使用批量大小为8，上下文大小为4，滑动步长为4的设置创建数据加载器\n",
    "dataloader = create_dataloader_v1(raw_text, batch_size=8, max_length=4, stride=4, shuffle=False)\n",
    "\n",
    "# 获取数据加载器的迭代器\n",
    "data_iter = iter(dataloader)\n",
    "\n",
    "# 获取第一个批次的输入和目标\n",
    "inputs, targets = next(data_iter)\n",
    "\n",
    "# 打印输入和目标\n",
    "print(\"Inputs:\\n\", inputs)\n",
    "print(\"\\nTargets:\\n\", targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cd2fcda-2fda-4aa8-8bc8-de1e496f9db1",
   "metadata": {},
   "source": [
    "## 2.7 创建词元嵌入"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a301068-6ab2-44ff-a915-1ba11688274f",
   "metadata": {},
   "source": [
    "- 数据已经几乎准备好用于大型语言模型（LLM）\n",
    "- 但最后，让我们使用嵌入层将词元嵌入到连续的向量表示中\n",
    "- 通常，这些嵌入层是大型语言模型的一部分，并且在模型训练期间进行更新（训练）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e85089aa-8671-4e5f-a2b3-ef252004ee4c",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/15.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44e014ca-1fc5-4b90-b6fa-c2097bb92c0b",
   "metadata": {},
   "source": [
    "- 假设我们有以下四个输入示例，分词后的输入ID为2、3、5和1："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "15a6304c-9474-4470-b85d-3991a49fa653",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 分词后的输入ID示例\n",
    "input_ids = torch.tensor([2, 3, 5, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14da6344-2c71-4837-858d-dd120005ba05",
   "metadata": {},
   "source": [
    "- 为了简单起见，假设我们只有6个词的小词汇表，并且我们希望创建大小为3的嵌入向量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "93cb2cee-9aa6-4bb8-8977-c65661d16eda",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义词汇表大小为6，嵌入维度为3\n",
    "vocab_size = 6\n",
    "output_dim = 3\n",
    "\n",
    "# 设置随机种子以确保结果可复现\n",
    "torch.manual_seed(123)\n",
    "\n",
    "# 创建嵌入层，词汇表大小为6，嵌入维度为3\n",
    "embedding_layer = torch.nn.Embedding(vocab_size, output_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ff241f6-78eb-4e4a-a55f-5b2b6196d5b0",
   "metadata": {},
   "source": [
    "- 这将生成一个 6x3 的权重矩阵："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "a686eb61-e737-4351-8f1c-222913d47468",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameter containing:\n",
      "tensor([[ 0.3374, -0.1778, -0.1690],\n",
      "        [ 0.9178,  1.5810,  1.3010],\n",
      "        [ 1.2753, -0.2010, -0.1606],\n",
      "        [-0.4015,  0.9666, -1.1481],\n",
      "        [-1.1589,  0.3255, -0.6315],\n",
      "        [-2.8400, -0.7849, -1.4096]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "# 打印嵌入层的权重矩阵（6x3）\n",
    "print(embedding_layer.weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26fcf4f5-0801-4eb4-bb90-acce87935ac7",
   "metadata": {},
   "source": [
    "- 对于熟悉独热编码的人来说，上述嵌入层的方法本质上是一种更高效的实现方式，相当于在全连接层中进行独热编码后进行矩阵乘法的操作，这在补充代码 [./embedding_vs_matmul](../03_bonus_embedding-vs-matmul) 中有描述\n",
    "- 由于嵌入层只是独热编码和矩阵乘法方法的更高效实现，因此它可以被视为一个可以通过反向传播优化的神经网络层"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b0d58c3-83c0-4205-aca2-9c48b19fd4a7",
   "metadata": {},
   "source": [
    "- 要将ID为3的词元转换为3维向量，我们执行以下操作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "e43600ba-f287-4746-8ddf-d0f71a9023ca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.4015,  0.9666, -1.1481]], grad_fn=<EmbeddingBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# 将ID为3的词元转换为3维嵌入向量\n",
    "print(embedding_layer(torch.tensor([3])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7bbf625-4f36-491d-87b4-3969efb784b0",
   "metadata": {},
   "source": [
    "- 请注意，上述结果是 `embedding_layer` 权重矩阵的第4行\n",
    "- 要嵌入之前定义的所有 `input_ids` 值，我们执行以下操作："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "50280ead-0363-44c8-8c35-bb885d92c8b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 1.2753, -0.2010, -0.1606],\n",
      "        [-0.4015,  0.9666, -1.1481],\n",
      "        [-2.8400, -0.7849, -1.4096],\n",
      "        [ 0.9178,  1.5810,  1.3010]], grad_fn=<EmbeddingBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# 对所有输入ID进行嵌入操作\n",
    "print(embedding_layer(input_ids))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be97ced4-bd13-42b7-866a-4d699a17e155",
   "metadata": {},
   "source": [
    "- 嵌入层本质上是一种查找操作："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f33c2741-bf1b-4c60-b7fd-61409d556646",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/16.webp?123\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08218d9f-aa1a-4afb-a105-72ff96a54e73",
   "metadata": {},
   "source": [
    "- **你可能对比较嵌入层与常规线性层的附加内容感兴趣：[../03_bonus_embedding-vs-matmul](../03_bonus_embedding-vs-matmul)**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c393d270-b950-4bc8-99ea-97d74f2ea0f6",
   "metadata": {},
   "source": [
    "## 2.8 编码词的位置"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24940068-1099-4698-bdc0-e798515e2902",
   "metadata": {},
   "source": [
    "- 嵌入层将ID转换为相同的向量表示，而不考虑它们在输入序列中的位置："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e0b14a2-f3f3-490e-b513-f262dbcf94fa",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/17.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92a7d7fe-38a5-46e6-8db6-b688887b0430",
   "metadata": {},
   "source": [
    "- 位置嵌入与词元嵌入向量相结合，形成大型语言模型的输入嵌入："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48de37db-d54d-45c4-ab3e-88c0783ad2e4",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/18.webp\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f187f87-c1f8-4c2e-8050-350bbb972f55",
   "metadata": {},
   "source": [
    "- BytePair 编码器的词汇表大小为 50,257：\n",
    "- 假设我们希望将输入词元编码为 256 维的向量表示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "0b9e344d-03a6-4f2c-b723-67b6a20c5041",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 定义词汇表大小为50257，嵌入维度为256\n",
    "vocab_size = 50257\n",
    "output_dim = 256\n",
    "\n",
    "# 创建词元嵌入层，词汇表大小为50257，嵌入维度为256\n",
    "token_embedding_layer = torch.nn.Embedding(vocab_size, output_dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2654722-24e4-4b0d-a43c-436a461eb70b",
   "metadata": {},
   "source": [
    "- 如果我们从数据加载器中采样数据，我们会将每个批次中的词元嵌入为 256 维向量\n",
    "- 如果批量大小为 8，每个批次包含 4 个词元，这将生成一个 8 x 4 x 256 的张量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "ad56a263-3d2e-4d91-98bf-d0b68d3c7fc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 设置上下文最大长度为4\n",
    "max_length = 4\n",
    "\n",
    "# 使用批量大小为8，最大长度为4，步长为4的设置创建数据加载器\n",
    "dataloader = create_dataloader_v1(\n",
    "    raw_text, batch_size=8, max_length=max_length,\n",
    "    stride=max_length, shuffle=False\n",
    ")\n",
    "\n",
    "# 获取数据加载器的迭代器\n",
    "data_iter = iter(dataloader)\n",
    "\n",
    "# 获取输入和目标数据\n",
    "inputs, targets = next(data_iter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "84416b60-3707-4370-bcbc-da0b62f2b64d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token IDs:\n",
      " tensor([[   40,   367,  2885,  1464],\n",
      "        [ 1807,  3619,   402,   271],\n",
      "        [10899,  2138,   257,  7026],\n",
      "        [15632,   438,  2016,   257],\n",
      "        [  922,  5891,  1576,   438],\n",
      "        [  568,   340,   373,   645],\n",
      "        [ 1049,  5975,   284,   502],\n",
      "        [  284,  3285,   326,    11]])\n",
      "\n",
      "Inputs shape:\n",
      " torch.Size([8, 4])\n"
     ]
    }
   ],
   "source": [
    "# 打印词元ID\n",
    "print(\"Token IDs:\\n\", inputs)\n",
    "\n",
    "# 打印输入数据的形状\n",
    "print(\"\\nInputs shape:\\n\", inputs.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "7766ec38-30d0-4128-8c31-f49f063c43d1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 4, 256])\n"
     ]
    }
   ],
   "source": [
    "# 将输入词元通过嵌入层映射为嵌入向量\n",
    "token_embeddings = token_embedding_layer(inputs)\n",
    "\n",
    "# 打印嵌入向量的形状\n",
    "print(token_embeddings.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe2ae164-6f19-4e32-b9e5-76950fcf1c9f",
   "metadata": {},
   "source": [
    "- GPT-2 使用绝对位置嵌入，因此我们只需创建另一个嵌入层："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "cc048e20-7ac8-417e-81f5-8fe6f9a4fe07",
   "metadata": {},
   "outputs": [],
   "source": [
    " # 上下文长度设置为最大长度\n",
    "context_length = max_length \n",
    "# 创建位置嵌入层，输入维度为上下文长度，输出维度为输出维度\n",
    "pos_embedding_layer = torch.nn.Embedding(context_length, output_dim)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "c369a1e7-d566-4b53-b398-d6adafb44105",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([4, 256])\n"
     ]
    }
   ],
   "source": [
    "# 通过位置嵌入层获取位置嵌入，输入为从0到max_length的索引\n",
    "pos_embeddings = pos_embedding_layer(torch.arange(max_length)) \n",
    "\n",
    "# 打印位置嵌入的形状\n",
    "print(pos_embeddings.shape)  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "870e9d9f-2935-461a-9518-6d1386b976d6",
   "metadata": {},
   "source": [
    "- 要创建在 LLM 中使用的输入嵌入，我们只需将词元嵌入和位置嵌入相加："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "b22fab89-526e-43c8-9035-5b7018e34288",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([8, 4, 256])\n"
     ]
    }
   ],
   "source": [
    "# 将词元嵌入和位置嵌入相加，生成输入嵌入\n",
    "input_embeddings = token_embeddings + pos_embeddings  \n",
    "\n",
    "# 打印输入嵌入的形状\n",
    "print(input_embeddings.shape)  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fbda581-6f9b-476f-8ea7-d244e6a4eaec",
   "metadata": {},
   "source": [
    "- 在输入处理工作流的初始阶段，输入文本被分割成单独的词元\n",
    "- 在此分割之后，这些词元根据预定义的词汇表被转换为词元ID："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1bb0f7e-460d-44db-b366-096adcd84fff",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch02_compressed/19.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63230f2e-258f-4497-9e2e-8deee4530364",
   "metadata": {},
   "source": [
    "# Summary and takeaways"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b3293a6-45a5-47cd-aa00-b23e3ca0a73f",
   "metadata": {},
   "source": [
    "查看 [./dataloader.ipynb](./dataloader.ipynb) 代码笔记本，这是我们在本章中实现的数据加载器的简洁版本，接下来训练 GPT 模型时将需要它。\n",
    "\n",
    "请查看 [./exercise-solutions.ipynb](./exercise-solutions.ipynb) 获取练习的解决方案。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05d923c7-b521-46b4-b404-a692b14352d1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
